Equal Bits: Enforcing Equally Distributed Binary Network Weights

Binary networks are extremely efficient as they use only two symbols to define the network: {+1, −1}. One can make prior distribution of these a design choice. The recent IR-Net Qin et al. argues that imposing Bernoulli with equal priors (equal bit ratios) over binary weights leads maximum entropy and thus minimizes information loss. However, work cannot precisely control weight during training, therefore guarantee entropy. Here, we show quantizing using optimal transport any ratio, including ratios. We investigate experimentally ratios indeed preferable our method optimization benefits. quantization is effective when compared state-of-the-art binarization methods, even pruning. Our code available at https://github.com/liyunqianggyn/Equal-Bits-BNN.